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1. Introduction 

The purpose of this paper is to propose an efficient method to compute the 
automorphism group of an arbitrary hyperelliptic function field over a given ground 
field of characteristic > 2 as well as over its algebraic extensions. Beside theoretical 
applications, knowing the automorphism group of a hyperelliptic function field also 
is useful in cryptography: 

The Jacobians of hyperelliptic curves have been suggested by Koblitz as groups 
for cryptographic purposes, because the computation of the discrete logarithm 
is believed to be hard in this kind of groups ([Kob89]). In order to obtain "se- 
cure" Jacobians it is necessary to prevent attacks like Pohlig/Hellman's ([PH78]), 
Frey/Ruck's ([FR94]) and Duursma/Gaudry/Morain's ([DGM99]). The latter at- 
tack is only feasible, if the corresponding function field has an automorphism of 
large order. To forestall the Pohlig-Hcllman attack, one needs to assert that the 
group order is almost prime, i.e. it ought to contain a large prime factor pq. To 
prevent the Frey-Riick attack, po needs to possess additional properties. 

Therefore, one needs to know both the automorphism group of the function field 
and the order of the Jacobian. Unfortunately, there is no efficient algorithm known 
to compute this order for arbitrary hyperelliptic curves. Only for specific types of 
curves, divisor class counting 1 is feasible for cryptographically relevant group sizes 
(e.g. [SSI98], [GHOO]). 

A theorem by Madan ([Mad70]) implies that |Jf| divides \If'\ whenever F C 
F' is a (hyper-)elliptic subfield of a hyperelliptic function field s.th. [F' : F] < oo. 
Thus, a hyperelliptic function field with secure Jacobian will most likely have a 
trivial automorphism group, i.e. one consisting of the hyperelliptic involution, only. 
Therefore, the proposed technique provides a quick test to check whether a given 
hyperelliptic curve may yield a secure Jacobian, i.e. whether it is worthwhile to 
apply expensive divisor class counting algorithms. 

Let us outline the afore mentioned algorithm briefly. It is well known that the 
automorphism group of a hyperelliptic function field is finite (cf. [Sch.38]). For each 
finite group, which can occur as subgroup of such an automorphism group, Brandt 
gave a normal form for the corresponding hyperelliptic function fields and explicit 
formulas for these automorphisms (cf. [Bra88]). Brandt's results only apply to 
function fields over algebraically closed constant fields, but this is no hindrance as 
we will see later. For now, we suppose the constant field to be algebraically closed. 

Hence, computing the automorphism group reduces to the problem of deciding, 
whether a given hyperelliptic function field has a defining equation of the form given 
by Brandt's theorems. This can be checked using theorem 10, which states that 
two hyperelliptic function fields k(t,u), k(x,y) with u 2 = Dt, y 2 — D x are equal 
iff x — ^rfr^r f° r some ccj £ k and y — tpu, where <p £ k(t) can be determined 
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from the a; t . Hence, we substitute x = ^It+al symbolically into D x . Computing ip 
according to the theorem and comparing coefficients of D t and tp~ 2 £> a ( ^*^"^ ) = 
<p~ 2 y 2 = u 2 = D t , we obtain a system of polynomial equations for the on. These 
can be tested for solvability or even solved using Grobner basis methods. 

If the constant field fc is algebraically closed, algorithm 7 seems to be the only 
efficient possibility known to compute the automorphism group of an arbitrary 
hyperelliptic function field. For finite fc, the method described in section 4.2 is an 
alternative approach to the AutomorphismGroup function in [StoOl]. 

2. Notation and Fundamental Facts 

Throughout this paper, we use the notations from [Sti93] . For the reader's 
convenience, we recall the essential notations: The natural numbers N start at 0, 
N+ := N \ {0}. The greatest common divisor of two integers or polynomials p, q is 
denoted by (p, q). The unit group of a field k is denoted by k* := k\ {0}. Let k 
be some field of characteristic p > 2, and g G N, g > 1. A hyperelliptic function 
field of genus g over k is defined to be a field F := k{x, y) s.th. x is transcendental 
over k and y 2 = D(x) 1 where D G k[x] is a monic separable polynomial of degree 
2g + 1 or 2g + 2. The automorphism group of F is the group Aut(F/fc) of field 
automorphisms of F fixing k. If U < Aut(F/k), we denote the fixed field of U by 
F u . The algebraic closure of k is denoted by k. If P is a place of F, vp denotes 
the valuation corresponding to P. For t G F we denote the principal divisor of t by 
(t), its zero divisor by (t) and its pole divisor by (t)^. If (t)^ is a place, we also 
denote it by oo t := (t)^ and call it the infinite place w.r.t. t. 

Our aim is to compute the automorphism group of any given hyperelliptic func- 
tion field k(x,y), y 2 = D. As mentioned above, Brandt gives normal forms of 
hyperelliptic function fields for each possible finite subgroup of the automorphism 
group (cf. Brandt's Ph.D. thesis, [Bra88]). Since the automorphism group of such a 
field is a central extension of Aut(fc(a;)/fc) by the C 2 generated by the hyperelliptic 
involution, Brandt rather investigates the possible subgroups of Aut(fc(x, y)/k)/C2, 
i.e. he characterizes the fields by their "type" which is defined as follows. 

Definition 1. Type of fieldF[G, k] Let F/fc be a hyperelliptic function field and 
G some finite group. F is called a function field of type F[G, fc], if there arc finite 
groups G, U, s.th. U < Aut(F/fc), C < U, C = C 2 , F c is a rational function field 
over fc, and U/C ^ G. 

We denote such a group U by U(G) or Uf(G), although U needs not to be 
uniquely determined by F, G and fc. We will only use this notation to state that a 
specific group can be used as U in this definition. 

For extension fields fc' D fc, we call F to be of type F[G, fc'] iff the constant field 
extension Fk'/k' is of type F[G, fc']. 

The following types can occur for hyperelliptic function fields over algebraically 
closed constant fields of characteristic p > 2: F[C„,fc], where (n,p) = 1, F[C™,fc] 
for some m G N+, F[£>„,fc], where (n,p) — 1 or n — p, F[^4,fc], FL4s,fc], F[c>4,fc], 
F[C™ xC„,fc], where (n,p) — 1 and m G N+, F[PSL 2 (p m ), fc], where m G N+ 
and F[PGL 2 (p m ), fc], where m G N+. As one needs to consider several cases for 
some of these types, the theorem stating Brandt's normal forms contains 14 case 
distinctions. For brevity, we only consider the types with the smallest and largest 
possible subgroups as well as a subgroup which we will need in our examples. 
Hence, we restrict ourselves to the types F[C„,fc], where (n,p) — 1, F[C™,fc] for 
some m G N+, F[2?„,fc], where (n,p) = 1, and F[PGL 2 (p m ), fc], where m G N+. 
The remaining cases are similar to these and can be found in [G6bon] or directly 
in [Bra88]. 
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Theorem 1 (Brandt). Let F be a hyperelliptic function field over an algebraically 
closed constant field k of characteristic p > 3. Then the types of F are characterized 
as follows 

(1) F is of type F[C„, k] for n £ N + with (n,p) = 1 iff there are t,u £ F, s.th. 
F = k{t,u), u 2 = t 1/ f\ s j=1 {x n -a j ), where v £ {0, 1}, s £ N and the aj £ k* 
are pairwise distinct. 

In this case, Uf(C„) is generated by ip : 1 i— » t, u i— > — u and ip : 1 1— » n 2 t, 
u i ► r^u, w/iere n is a primitive 2n-th root of unity. 

(2) F is of type F[C™,fc] wit/i m £ N+ iff there are t, u £ F and a sub- 
group A of the additive group of k of order \A\ — p m , s.th. F = k(t,u), 
u 2 = Y\ S j = i (YlaeA( x + a ) ~ a j)j where s £ N and the dj £ k are pairwise 
distinct. 

In this case, Up(C™) is generated by tp : t i— > t, mh — u and all ip a :t >—> 
t + a, ?i u with a £ A. 

(3) _F is of type F[T> n ,k], where n £ N+, (n,p) = 1 iff there are t,u £ F, 
s.th. F = k(t,u), u 2 = t v °{t n ~ \)^{t n + 1)' /2 \[ s j=1 {t 2n - a 3 t n + 1), where 
fj £ {0, 1}, s £ N and the aj £ k \ {±2} are pairwise distinct. If n = 2 or 
n = 1 mod 2, we need to have V\ = v^. 

In this case, Xip{T> n ) is generated by ip : t i— » t, u >—> —u, ip : t i— > f7 2 t, 
u i > ry^u and cr : t i— » \, u i— > 1 jsr, w/iere n is a primitive 2n-th root of 
unity, i 2 = —1 and m = \n(u\ + 1/2) + 2^o + 2ns. 
(14) F is of type F[PGL 2 (p m ), k] iff there are t,u e F, s.th. F = k(t,u), 

u 2 = {t r -t) va {{t r -ty- 1 + \) V1 

f[(({t r -ty- 1 + i) r+1 - aj (f -ty 2 - r ) , 

where Vj £ {0, 1}, s £ N, r = p m and the aj £ k* are pairwise distinct. 

In this case, UF(PGL 2 (p m )) is generated by ip : t 1— > t, mh — u, ip : t t-^> 
n 2 t, u 1 ^ n'^it, a : t ^ t + 1, u 1— > u and t:(^j,«h^, waere n is a 
primitive 2(p m — l)-th root of unity and 

n= l - (( P m + l)v a +p m (p m - l)v\ +p rn (p 2m - 1) 8 ) . 

Proof. A slightly more general theorem is proved by Rolf Brandt in his Ph.D. thesis 
[Bra88]: He characterizes the types of cyclic extensions of rational function fields 
over algebraically closed constant fields. We list the references for each of the 
stated facts, citing the proof that a function field of the given type has the given 
normal form, first. The proof of the inverse implication and the generators are 
given thereafter. 

(1) [Bra88, Satz 5.1], [Bra88, Satz 5.6] and [Bra88, Lemma 5.5]. 

(2) [Bra88, Satz 6.3] and its proof. 

(3) Cf. [Bra88, Satz 7.3], [Bra88, Satz 7.5] and [Bra88, Lemma 7.4], in the case 
n = mod 2. Otherwise, we apply [Bra88, Satz 7.9], as p > 3 and (n,p) = 
1 obviously imply (2n,p) = 1. The generators and the inverse implication 
are proved analogously to [Bra88, Satz 7.5] and [Bra88, Lemma 7.4]. 

(14) [Bra88, Satz 13.1], [Bra88, Satz 13.6] and [Bra88, Lemma 13.2]. 

□ 

Let us illustrate this theorem and the related problems with an example. 
Example 2. We consider F := ¥t(x, y), 

y 2 = x 5 + x 3 + x = x(x + 2)(x - 2){x + 3) (a; - 3) = x(x 2 - 4)(x 2 - 2). 
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Obviously F is of type F[C 2 , F7]. The basis x, y of F is not uniquely determined by 
F, neither is the defining equation. Therefore, we cannot immediately see if F is 
of any other types. 

In the following section, we solve this problem, i.e. we propose an efficient pos- 
sibility to find out, if a hyperelliptic function field has a given normal form. 

3. Relations Between Bases 

In this section we show the connection between different bases of a hyperelliptic 
function field (cf. theorem 10): If k(t,u) — k(x,y) is a hyperelliptic function field, 
then x needs to be a fraction of linear polynomials in t and the relation between u 
and y can be computed easily from these polynomials. In contrast to theorem 1, we 
do not need to have an algebraically closed constant field, here; theorem 10 applies 
to hyperelliptic function fields over arbitrary constant fields of characteristic 7^ 2. 
This theorem is one of the core components of our algorithm for computing the 
automorphism group of a hyperelliptic function field, as we will see in section 4. 

3.1. Relations Between the Variable Symbols. Here, we show that x can be 
represented as a fraction of linear polynomials in t. We start our proof by citing 
the following lemma: 

Lemma 2. Let k(t, u) — k(x, y) be a hyperelliptic function field, u 2 = D t , y 2 = D x , 
where D t G k[t] and D x G k[x] are separable monic polynomials. Then k(t) = k(x). 

Proof. [Sti93, Proposition VI.2.4]. □ 

Lemma 2 means, that the following proposition can be applied to our situation, 
i.e. in hyperelliptic function fields with two given bases, we always have k(x) = k(t). 
We see that a; is a fraction of linear polynomials in t in this case: 

Proposition 3. Let k(t) be a rational function field and x G k(t) s.th. k(t) = k(x). 
Then there are ao, ■ • • , a 3 G k with x — ^ll+T an< ^ a o a 3 — a i a 2 ^ 0. 

Proof. As x G k(t), there are polynomials ip, ip G k[t], s.th. x = £ and (<p,tp) G k. 
We consider the principal divisor of x. [Sti93, Theorem 1.4.11] implies 

deg(Or)o) = degdx)^ = [k(t) : k(x)} = 1. 

Let us consider the case oo* ^ supp(x) , first. Then = Voo t {x) = deg t (ip) — deg t (ip) , 
i.e. deg t (^) = deg t (tp). As ip,ip G k[t], we get (ip)^ = deg t (ip)oo t = deg t O)oo t = 
(^) oc . We have (x) = {<p) - ty) = (<p) (p) x (ty) - (VU = Mo - Wo, i.e. 

O)o = (V)o and ( x )oc = Wo- Thus > 

deg t (» = deg((^) oc ) = deg((ip) ) = deg((x) ) 

= 1 = deg((x) oc ) = deg((V>) ) = deg((V') oc ) = deg t (ip). 

Thus there are on G k s.th. ip = a t + ot\, tjj = a 2 t + a 3 and aoa^ — aia 2 7^ as 
claimed. 

If oo t G supp(x), we obviously have deg t (p) ^ dcg t (?/>)• W.l.o.g. we assume 
Voc t (x) < (consider - in the other case). As deg((x) oc ) = 1, we need to have 
Voo t (x) = -1. Thus 

-1 = Voo t (a;) = deg t (V>) - deg t (</>), 
i.e. deg t (tp) = deg t (<p) - 1. As (ip)^ = dcg t (^)oo t and (tp)^ = dcg t (»oo t , we infer 

(x) = {ip) - 0/0 = (<p) - Moo - Wo + Woe = Mo - Wo - °°t- 
Thus, we have (x) = (p) Q , i.e. 

deg t (<p) = deg((^) oc ) = deg(O)o) = deg((x) ) = 1. 
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Furthermore deg t (V>) = deg t (ip) — 1 = 0. We obtain x 
and aoQ!3 7^ as claimed. 



- with oil £ k 
□ 



Summing up these facts, we obtain, that a; is a fraction of linear polynomials 
in t, if k(x,y) = k{t,u): 

Corollary 4. Let k(t,u) = k(x,y) be a hyperelliptic function field, u 2 = D t , 
y 2 = D x , where D t £ k[t] and D x £ k[x] are separable monic polynomials. Then 
there are a , . . . ,a 3 E k with x = and a a 3 — «ia 2 ^ 0. 

Proof. By lemma 2, we have k{t) — k(x). Thus proposition 3 implies the existence 
of the oti. □ 

3.2. Relation Between the Square Roots. Since we know, how t and x are 

related in a hyperelliptic function field for which we have two bases k(t, u) — k(x, y), 
we proceed studying the relationship between u and y. The next lemma tells us, 
that y is a multiple of u over k{t): 

Lemma 5. Let F = k(t,u) — k(x,y), u 2 — D t , y 2 = D x be a hyperelliptic function 
field over a constant field k of characteristic ^ 2, where both D t £ k[t] and D x £ k[x] 
are monic separable polynomials. Then there is some ip E k(t) \ {0}, s.th. y = <pu. 

Proof. As y £ F — k(t,u) and [k(t, u) : k{t)\ = 2, there are tp,ip £ k(t) s.th. 
y = ipu + -0. Let us suppose ip — 0. Then we had y G k{t). From lemma 2 
we know that k(x) — k(t). Thus we had y £ i.e. k(x,y) = k(x) implying 

[k(x,y) : k(x)} = 1, which contradicts [k(x,y) : k(x)\ = 2. Therefore (p ^ 0. 
Substituting our representation of y into its minimal polynomial we get 

D x =y 2 = (ipu + ip) 2 = (p 2 u 2 + 2ipipu + tp 2 = ip 2 D t + 2<p>i\)u + ip 2 . 

Thus 2ip\pu £ k(t) — k(x). As u ^ k(t), this leads to 2ipij) — 0, from with we 
conclude ip = because char(fc) ^ 2 and (fi ^ 0. □ 

Knowing that y = ipu, we will examine if more closely. We start by the following 
lemma, which is quite technical, but will be useful in the subsequent proofs. 

Lemma 6. Let k(t, u) — k(x, y), u 2 — D t , y 2 — D x be a hyperelliptic function field 
over a constant field k of characteristic ^ 2, where both Dt £ k[t] and D x £ k[x] 
are monic separable polynomials. Let x — , on £ k, a^a^, — a\a 2 ^ as 

stated in corollary 4 an d V = i P u , f £ k(t) \ {0} as in lemma 5. Then there are 
d x := deg x (D x ) pairwise relatively prime pi £ k[t], deg t (pi) < 1 s.th. 



Furthermore we have 

(1) pi = (a + a 2 rji)t + ct\ — a 3 r/i, where r\i £ k, i = 1, . . . , d x are the zeroes of 



D t = <P~ 2 (a 2 t + a 3 y dx Y[pi 




(3) Let q £ k[t] be linear. Then q 2 /filial Pi- In particular, (a 2 t+a3) 2 ^Jjf^pi. 
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Proof. We factor D x over k into D x = rij=i( x — Vi)i Vi ^ k, r]i ^ r]j for all i ^ j. 
This yields 

D t =u 2 = <p~ 2 y 2 = <fi~ 2 D x = tp~ 2 Y[(x - f]i) 



i=i 



aot + ai 

i 



x a 2 t + a 3 



- Vi) 



M a 2 i + a 3 

-2 TT Pi 



a 2 t + a 3 

«=i 

=95- 2 (a 2 t + a 3 )-^l ; lft G fc[t], 

where := (ao — a.?x\i)t + ct\ — a^rji. Suppose there were indices i / j s.th. pi 
and pj had a common divisor of nonzero degree w.r.t. t. Then we had Pi = j3pj 
for some (3 G * \ {0}, i.e. (z - w ) = = = f3(x - Vj ). Thus, D a 

were not separable. Contradiction. Therefore, the pi are pairwise relatively prime, 
which proves our main claim. 

Let us proceed by examining the supplementary statements. Obviously, 

deg t ^11^ - dx - 

If dcg t (nil Pi) 

< d x - 1, there were two indices i ^ j s.th. Pi,Pj G k, thus 

ao — 012m = cto — ot2"m = 0, i.e. ao = d-if]i — OLiVj- Hence, a2(r]i — rjj) = which 
yields 012 = since rji ^ rjj. Now we can easily deduce ao — from ao — a2t]i = 0. 

Since aoa 3 — aia2 7^ 0, this is not possible. Thus deg t (nf=iPi) — d x ~ 1- 
Finally, let q v \ \Yi=\Pi for 

some linear q G A;[t] and v G N+. As deg 4 (pi) < 1, 
there are v factors p^ , . . . , pt u , which are multiples of Thus p i± ,. . . , p iu are scalar 
multiples of each other. If v > 1, this contradicts the relative primality of the pi. 
This proves the last claim. □ 

The following lemma states, that ip^ 1 is a non-zero polynomial in t: 

Lemma 7. Let k(t, u) = k(x, y), u 2 = D t , y 2 = D x be a hyperelliptic function field 
over a constant field k of characteristic ^ 2, where both D t G k[t] and D x G k[x] 
are monic separable polynomials. Let x = °°^|°^ , a, G k, aoa 3 — aia 2 ^ as 
stated in corollary 4 and y — ipu as in lemma 5. Then we have ip^ 1 G k[t] \ {0}. 

Proof. Lemma 6 implies Dt = ip~ 2 (ot2t + a 3 )~ dx Ili=i-Pi- Suppose ip^ 1 — ^ ^ k[t]. 
As D t G k[t], ifiQ needs to be canceled by rii=iPi- Let q G k[t] be a linear factor 
of ip . Thus q 2 j nf=iPi> which contradicts lemma 6. Thus we need to have 
ip- 1 g k[t]. □ 

We will prove now, that Lp~ x is a power of the denominator of x, multiplied by 
some constant from k. 

Lemma 8. Let k(t, u) = k(x, y), u 2 — Dt, y 2 — D x be a hyperelliptic function field 
over a constant field k of characteristic ^ 2, where both D t G k[t] and D x G k[x] 
are monic separable polynomials. Let x = , a, G k, a a 3 — aia 2 ^ as 
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stated in corollary 4 and y = ipu as in lemma 5. Then there are 7 G k* and m G N 
s.th. 

(p- 1 = j(a 2 t + a 3 ) m . 

Proof. By lemma 7 we know ip^ 1 G k[t] \ {0}. Factoring it over k yields (p^ 1 = 
7 • (a 2 t + a 3 ) m , where 7 e k[t] \ {0} s.th. (a 2 t + a 3 ) 1 7 (7 does not need to be 
irreducible). By lemma 6 we have 

d x d x 

D t = ^ 2 (a 2 t + as)- 4 " \[pi = J 2 (a 2 t + a 3 ) m - d * f[ Pi . 

i=l i=l 

As D t is separable, we need to have 7 G k* which proves our claim. □ 

Computing the degree of <p _1 , we see that it is a scalar multiple of the (g + l)-th 
power of the denominator of x. 

Lemma 9. Let k(t, u) = k(x, y), u 2 = Dt, y 2 = D x be a hyperelliptic function field 
over a constant field k of characteristic ^ 2, where both D t G k[t] and D x G k[x) 
are monic separable polynomials. Let x — ^ft+^l > V ~ ^ u as s ^ e d in corollary 4 
and lemma 5. Then we have 

(1) If x e k[t], then ip G k* . 

(2) If x £ k[t], we assume w.l.o.g. a 2 = 1. Then there exists some 7 G k* s.th. 
if- 1 =7(t + a 3 )9 +1 . 

Proof. By lemma 6, there are pi G k[t], s.th. D t — <p~ 2 (a 2 t + a 3 )^ dx Yli=iPi an d 
d x — 1 < dcg t ^nf=iP») — d x - Let us consider the given cases, separately. 

(1) Let us assume x G k[t], first. We already know ip -1 G fc[i]\{0} (cf. lemma 7) 
and D t = p>^ 2 a z d,c Yli=i Pi- ^ f^ 1 ^> then ip~ 2 were a non trivial square 
polynomial in t dividing D t . This contradicts the separability of D t . Thus 
tp~ l G k, which immediately implies ip G k* . 

(2) We proceed with the case x £ k[t], i.e. a 2 ^ 0. By reducing the fraction 
x = 2° 2 t+al ' we can assume a 2 — 1 without loss of generality. As <p~ x G k[t], 
we get 



\i=l 



deg t (A) =2deg t (¥> L ) - d x deg t (i + a 3 ) + deg t \J\pi 
=2dcg t (<^" 1 ) - d x + deg t Y[pi , 



Vi=l 



which implies 

Sdeg^- 1 ) = de St (D t ) + d x - deg t (jl^ ■ 

Thus, the inequality — 1 < dcg t ^Of=i P») — d x yields 
degt (A) =deg t (A) + d x - d x 

< deg t (A) + d x - deg t J[ Pi 



\i=l 



=2deg t (<^- 1 ) 
<deg t (A) + c«x-rfx + l 
= deg t (A) + l- 
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Asdeg t (Z? t ) G {2g+l, 2g+2} we conclude deg t ((^ _1 ) = g+1. From lemma 8 
we know that there is some 7 G k* and some m G N s.th. p^ 1 = j(t + a 3 ) m . 
As deg^t/?" 1 ) = g + 1, this implies our claim. 

□ 



3.3. Putting Both Relations Together. The following theorem completely char- 
acterizes the relation between any two bases of a hyperelliptic function field of 
characteristic ^ 2. This can be used to check whether a given function field has 
a specific kind of defining equation. It is the key ingredient of algorithm 7, which 
computes automorphism groups. 

Using the facts proved above, it remains to compute the scalar factor of p in 
order to know the relation between two bases: 

Theorem 10. Let k(t,u) — k(x,y), u 2 = D t , y 2 = D x be a hyperelliptic function 
field over a constant field k of characteristic ^ 2, where both D t G k[t] and D x G k[x] 
are monic separable polynomials. Let d x :— dcg x (D x ). 

(1) If x E k[t], then there are a ,ai G k s.th. x — a t + ax and a ^ 0. 
Furthermore we have y = (pu with <p G k* , 

p 2 =a d °. 

(2) If x ^ k[t], then there are a a ,ai,a 3 £ k, s.th. x — ""l^ 1 , a a 3 — a\ ^ 0. 
Furthermore we have y — pu, where 

= 13 

with [3 G k* . For (3 we have the formula 



a 2 



D x (a ) ,ifD x (a )^0 
{ai - a a 3 )D x (a ) , if D x (a ) = 0, 



where D T (x] :— D = c ^ 



Proof. Corollary 4 gives the existence of ao,...,a 3 G k s.th. x — and 
0:00:3 — ot\ot2 7^ 0. Lemma 5 yields some <p G k(t) \ {0} s.th. y = pu. By lemma 9, 
we know p G k* if x G k[t] and ps^ 1 — j(a 2 t + a 3 ) 9+1 with 7 G k* otherwise. 
Lemma 6 implies 

D t = p~ 2 (a 2 t + a 3 )~ dx IJp»> (1) 

where pi = (a — a 2 r/i)t + ct\ — a 3 r\i and the 77, G k are the zeroes of D x . 
Let us consider the different cases, now: 

(1) If x G k[t], we have a 2 = 0. Reducing the fraction a "^ ai , we may w.l.o.g. 
assume 03 = 1. Thus equation (1) becomes 

dx 

D t = p~ 2 Y[{a t + ai - rii). 

As a ^ (which we conclude from 0:0^3 — aia 2 = a ^ 0) and (p G k, the 
leading coefficient of D t is 

1 = lc t (A) = f- 2 afr, 
because D t is monic by assumption. This implies p 2 = o dx . 
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(2) If x ^ k[t], we have a 2 ^ 0. Reducing the fraction , we may assume 

a 2 — 1. We already know ip^ 1 = j(t + a 3 ) 9+1 . Setting (3 := 7 _1 , it remains 
to compute 1 . From equation (1), we get 

As before, we compute the leading coefficients: 

i = ict(A) = ic t (^- 2 (t + ^? a+2 - d * J[p^j = r 2 k t (j[p)j . 

We obtain 

/3 2 = lc^fi^- (2) 

From Lemma 6, we know d x — 1 < deg t (nf=i Pi) — dx- Thus, there are two 
cases: deg t (nf=i Pi) — d x and dcg t (nf=iP«) = d x — 1. In the latter case, 
there is some index j s.th. pj = (ceo — Vj)t + a i — <= &i i- e - a o — % = 0. 
Hence, a a = r/j, which implies D x (a ) = 0. In the former case, there is no 
such index, i.e. we have D x (ao) ^= 0. 

(a) If D x (a ) ^ 0, we have a — r\i ^ for all i. Thus we get 

= Y[( a o - Vi) = D x {a ) 

i=l 

as claimed. 

(b) If D x (a ) = 0, there is exactly one index j s.th. r/j = a . W.l.o.g. we 
assume j — d x . Then we have pd x = a\ — a a 3 . Thus equation (2) 
implies 

1 =lc t ^11^ = lc * ^II( a o - Vi)t + ai- a 3 r/^j 

/ d x -l \ 

=lc t (ai - a a 3 ) ]J (a - 7ft )t + «i - "3^ 



=(ai - aoa 3 ) ]J (ao - 
i=i 

=(ai - a a 3 )L' a; (Q;o), 



□ 



Corollary 11. Letk(x,y), y 2 — D x be a hyperelliptic function field over a constant 
field k of characteristic ^ 2, where D x G k[x] is a monic separable polynomial. Let 
D t G k[T] be another monic separable polynomial. There exists a basis t,u G k(x, y) 
s.th. k(x, y) = k(t, u), u 2 = D t (t) iff there exist t,u G k(x, y) for which u 2 = D t (t) 
and the relations x = ; y — ipu given in theorem 10 hold. 
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Proof. It remains to show that the existence of t, u, u 2 = D t (t) s.th. x = , 
y = pu as given in theorem 10 implies k(x,y) — k(t,u). It is obvious, that k(x) C 
k(t) and k(t)(u) = k(t)(y). Solving x = %°*+^ for t, we see fe(i) C k(x). Thus 
fc(i) = fc(ar), i.e. k{t,u) = k{t){u) = k(t)(y) = k(x){y) = k(x,y). □ 



As said before, theorem 10 can be applied to check if a hyperelliptic function 
field k(x,y) has a basis t, u satisfying a given equation u 2 = D t . According to 
corollary 11, we can decide this question by checking, if there are a, £ fc, and 
ip G k(t) as given in theorem 10 s.th. u 2 = D u which is equivalent to D t = u 2 = 
p~ 2 y 2 = p~ 2 D Xl here. This can be done using the following algorithm: 

Algorithm 3. Let k(x,y), y 2 = D x , D x G k[x] monic and separable, be some 
hyperelliptic function field of genus g with char(fc) ^ 2 and let D t G k[t] be some 
monic, separable polynomial of deg t (D t ) G {2g + 1, 2g + 2}. Let d x := deg x (D x ). 

(1) We compute p 2 G k(t) symbolically from the on according to theorem 10. 
Since we do not know the a, in advance, we cannot tell which of the cases 
of our theorem applies. Thus we have to compute p 2 and do the following 
steps in each of these cases: 

• If x G k[t], we have to use x = a^t + a\, (p 2 = ct^ . 

• If x k[t], i.e. x = Q t °^" 1 , we consider both D x (a Q ) ^ and D x (a Q ) = 
0. In the former case we have p 2 = D x (ao)(t + as)~ 29+2 . If D x (ao) = 
0, we know that x — ao is a divisor of D x . Thus we can find all possible 
«o explicitly by factoring D x over k. For each such ao, we compute 
Ac : = |=S obtainin g ^ = («i - a a 3 )D x {a ){t + a 3 y 2 3- 2 . 

(2) After multiplying by the denominators, our condition D t = p~ 2 D x becomes 
an equation of polynomials in t and the ai. We compare coefficients of t. 
The resulting system of polynomial equations for the ai is denoted by (*). 

(3) Let the ideal I be generated by (*) and the polynomial 1 — {a$az — aia 2 )T, 
where T is new variable symbol and the ai satisfy x — °°*t ai according to 
the case we are considering. Using Grobner basis methods, we check I for 
solvability and construct a solution, if it exists. 

Thus we can construct a basis k{x,y) — k(t,u), u 2 = D t iff there are a, n T in the 
variety of I over k for any of the cases mentioned in step (1). 

Let us illustrate this algorithm with an example: 

Example 4. Let k = F n , F = k(x, y), y 2 = D x := x 5 + x 4 + 4x 3 + bx 2 + lQx + 7. 
We would like to know, if there is a basis F — k(t,u) s.th. u 2 = D t := t 5 + 7t 3 + 
9t 2 + 9t + 6. 

We start with the easiest case x G k[t]. From theorem 10, we get x = a t + a\, 
y = pu and p 2 — a$. 
Substituting, we get 

D x = D x (a t + ai ) =a 5 t 5 

+ (5a 4 ai + ao)* 4 

+ (lOalaj + Aalax + Aafyt 3 

+ (lOoiQcq + 6a Q a 1 + a Q ai + 5a )t 

+ (Haoa 4 + 4aoa 3 + aoa 2 + lOaoai + 10ao)t 

+ a\ + a\ + 4a? + 5aj + 10ai + 7 
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where cxq, ol\ are to be found. Comparing coefficients in D x = ip 2 D t — o^D t yields 
the equations (*): 

5Q!qQ;i + Qfg = 0, 

I0ala\ + ialca + 4a„ = 7al, 

lOagaf + 6alaf + a^cti + 5a^ = 9a^, 

5aoaf + 4a a^ + a a 2 + 10a oei + 10a = 9a§, 

a\ + a\ + Aa\ + ba\ + lOai + 7 = Qa\ 

Augmenting (*) by 1 — aoT, we get the ideal I. Singular ([GPS + 02]) computes the 
following Grobner basis of I w.r.t. the lexicographical ordering: 

T-4=0 

ao — 3 =0 

ai - 2a 5 T 2 - 3alT 2 =0 

This implies T = 4, ao = 3. Substituting these values into the remaining equation, 
we obtain ol\ = 2. Thus, setting t := Ax — 3, u := y, i.e. x = 3t + 2, (p = 3 = 1, we 
get a basis F = k(t, u), with u 2 = D t . 

In order to compute the automorphism group Aut(fc(x, y) / k) over an algebraically 
closed constant field, it suffices to check k(x, y) for normal forms, as we will see in 
section 4. This simplifies the Grobner basis step of algorithm 3, giving the following 
modified algorithm: 

Algorithm 5. Let k(x,y), y 2 = D x , D x e k[x] monic and separable, be some 
hyperelliptic function field of genus g with char(fc) 2 and Dt £ k[t] be some 
monic, separable polynomial of deg t (D t ) £ {2g + 1, 2g + 2}. Let d x := deg x (D x ). 

Whether there exists a basis k(x,y) = k(t,u) with u 2 — D t , can be checked 
analogous to algorithm 3. We only note the differences: 

(1) In order to compute ip 2 in the case x £ k[t], D x (ao) — 0, we have to consider 
all zeroes «o of D x over k, i.e. we have to factor D x over its splitting field. 

(3) Instead of constructing an element of the variety of /, we only need to check 
if it's empty To do so, we compute a Grobner basis B of I (e.g. w.r.t. the 
degree reverse lexicographical ordering). There exists a solution ol^T e k, 
iff 1+ (I), i.e. iff {I}. 

As in algorithm 3, we infer the existence of a basis k(x,y) = k(t,u), u 2 — D t iff 
B^{1}. 

Remark 6. An essential feature of algorithms 3 and 5 is, that D t does not need 
to be known completely. It may contain some parameters for which we can also 
solve. Therefore, we can use our algorithms to check, whether a given hyperelliptic 
function field k(x,y) has some of Brandt's normal forms (cf. theorem 1). We will 
see how to do this, in the following section. 

4. Computing the Automorphism Group 

4.1. Algebraically Closed Constant Fields. Algorithm 5 can be applied to 
compute the automorphism group of a hyperelliptic function field over an alge- 
braically closed constant field: 

Algorithm 7. Let k(x,y), y 2 = D x , D x G k[x] monic and separable, be a hyper- 
elliptic function field of genus g and char(fc) ^ 2. We denote F := k(x,y). 

(1) For each possible type F[G, k], we look up the corresponding normal form 
u 2 = D t in theorem 1. 
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(2) For each normal form found in step (1), we check, for which parameter sets 
D t has degree 2g + 1 or 2g + 2. This yields the set TV of all polynomials 
Dt, s.th. u 2 = Dt is a normal form for a field of genus g and type F[G, k\. 
The integer parameters in each Dt £ N are fixed, while the Dt may still 
contain parameters from k. 

(3) For each G, N and each fl f e JV, we check if F has a basis F = k(t, u) 
satisfying u 2 = D t as well as the additional conditions from theorem 1. To 
do so, we use a slight modification of algorithm 5: 

Let Co and C\ be the sets of polynomials that according to theorem 1 
have to be = and ^ 0, respectively. Let 

c := (o o 3 - «ia 2 ) /. 

fed 

We define the ideal I to be generated by (*), Co and 1 — c • T rather than 
just by (*) and 1 — (0:00:3 — otia 2 )T. Note that the polynomial ring RD I 
may contain more variables than just the on and T, now. 

We apply the rest of algorithm 5 without any changes. 

The variety of / is non-empty iff k(x, y) is of type F[G, k\. 

(4) Let G be the largest group G s.th. k(x,y) is of type F[G, k]. Then 

Aut(k(x,y)/k)/C 2 = G, 

and the generators of U(G) = Aut(fc(x, y)/k) are given in theorem 1. 

Thus, we are able to compute the structure as well as the generators of the 
automorphism group Aut(k(x , y) / k) for each hyperelliptic function field k(x,y). 

Example 8. Let F := ¥r(x, y) with y 2 = x 5 + x 3 + x as in example 2. The above 
algorithm yields that_F is of the types F[C 2 ,fV], F[C 3 ,F7], F[C 6 ,TV], Y[V 2 ,W^\, 
F[D 3 ,fV] and F[D 6 ,FV). 

To see, how the algorithm works, we consider parts of the proof that F is of type 
F[Z>3,Ft): 

(1) The normal form for fields of type F[£>3,Fy] is given by 

s 

y 2 = t v< >(t 3 - ir (t 3 + if 2 Y[(t 6 - aj t 3 + 1), 

i=l 

where Vi G {0, 1}, v\ = v 2 , s € N and the aj £ F7 \ {±2} are pairwise 
distinct. 

As g = 2, we need to have deg t (Z) t ) e {5, 6}, from which we get 

N = {(t 3 -l)(t 3 + l),t 6 - ai t 3 + l} 

Algorithm 5 finds out that F possesses a basis F = ¥7(t,u), u 2 = (t 3 — 
l)(t 3 + 1). Thus, F is of type F[X>3,Fy]. The corresponding system (*) 
of equations and inequalities is not given, as it looks quite ugly and does 
not help in understanding this step of the algorithm. It is similar to the 
one given in example 4. Simplifying (t 3 — l)(t 3 + 1) = t 6 — 1, theorem 1 
immediately implies that F is of type F[2? 6 ,F7]. 

Furthermore, the second clement of N can also be used to find a basis 
of F: Setting a\ := and oi := i, where i 2 = —1, implies 00 = 1 and 
o 3 = -i. Thus F = ¥^(v, w) with w 2 = v 6 + 1. 

From the list above, we know that Vq is the largest group G s.th. F is 
of type F[G,F 7 ]. Thus, Aut(F7(x, 2/VF7) is a central extension of X> 6 by 
the C 2 , generated by the hyperelliptic involution. From the normal form 
u 2 = (t 6 + 1), we know uq = v\ = s = 0, v 2 = 1. According to theorem 1, a 
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set of generators of Aut(F 7 (x, y)/F 7 ) is given by {ip, tp, c}, where <p : t i— ► t, 
u i— > —it, : t h> (t, u h> « and cr : i i— > i, u i— > ^ with a primitive 6-th 
root £ of unity. 

Looking at these generators, we conclude Aut(F 7 (x, y)/F 7 ) = Vq x C 2 . 

4.2. Arbitrary Constant Fields. Using algorithm 7, it is also possible to com- 
pute Aut(k(x,y)/k) for a hyperelliptic function field k(x,y), where k needs not to 
be algebraically closed. A similar application is the computation of the smallest 
algebraic extension k! D k s.th. Aut(k'(x,y)/k') = Aut (k(x,y)/k). 

Let k be any field of characteristic p > 2 and k(x,y) be a hyperelliptic function 
field. We use algorithm 7 to compute the types of k(x,y). Let k(x,y) be of type 
F[G, k] and let k(x, y) = k(t, u), u 2 = D t be the corresponding normal form. Solving 
the ideal I for the on and the parameters of D t , we obtain explicit formulas for the 
generators of U(G). Using these, it is easy to find the smallest field k' D k, s.th. all 
automorphisms from U(G) define automorphisms of k'(x,y). Then, k' D k is the 
smallest field extension s.th. k(x,y) is of type F[G, k']. 

This method is used to solve the two problems given above: In order to compute 
Aut(fc(x, y)/k), we construct k' for each G s.th. k(x,y) is of type F[G, k]. The 
largest G with k' = k yields U(G) = Aut(fc(x, y)/k). 

To find the smallest kl D k s.th. Aut(k' (x,y)/k') = Aut(fc(x, y)/k), we compute 
Aut(fc(x, y)/k) and construct k' for G = Aut(fc(x, y)/k)/C2 as explained above. We 
show how to apply this method in the following example. 

Example 9. We consider F := F 7 (x, y), y 2 = x 5 + x 3 + x, i.e. we examine the curve 
from example 8 over F 7 . We already know, that Aut(IV(x, y)/W?) = T> 6 x C 2 - Thus 
we set G := V 6 . To find out, for which extension k D F 7 we have Aut(fc(x, y)/k) = 
Aut(F 7 (x, y)/F 7 ) ~P 6 x C2, we have a closer look at the proof 2 that F 7 (x, y) is of 
type F[£>6,F 7 ]. As seen in example 8, we have k(x,y) = k(t,u), u 2 = t 6 + 1 and 
the automorphism group is generated by tp : t t— * t, ui— >— u, ip : t t— > Qt, u > u 
and a : t >— > |, u 1— > 7^, with a primitive 6-th root of unity £. As 3 is such a 6-th 
root, we may set £ := 3. Thus, our automorphism are defined over the smallest 
extension k 3 F 7 s.th. t,u £ k(x,y). 

Hence, to compute k we have to examine t and u more closely. They can be 
computed from x and y using the coefficients on from theorem 10. Therefore, k is the 
smallest field s.th. on 6 k. Solving the corresponding equations and inequalities, we 
get that x £ k[t], cto = 1, oi\ = i with i 2 = — 1 is a possible solution. Furthermore, 
there is no solution over F 7 . Thus, t,u £ F^^^y) := W?(x,y,i) which implies that 
k := F49 is the smallest constant field s.th. Fk — k(x, y) has the automorphism 
group T> 6 x C 2 - 

5. Computational Aspects 

The author implemented algorithm 7 for the computer algebra systems MuPAD 
([Sci02]) and Singular ([GPS + 02]). The Grobner basis steps are implemented for 
Singular, while anything else — i.e. Brandt's normal forms, computing N, substi- 
tution and the comparing of coefficients — is programmed for MuPAD. Both parts 
of the program are combined using shell scripts. It was decided to separate the 
Grobner basis steps from the rest of the computation, since on the one hand, Sin- 
gular has one of the most efficient Grobner basis implementations. On the other 
hand, Singular is restricted to characteristic p < 32003, which is too small for many 
fields of cryptographic relevance. 

As a proof of concept, the implementation is not optimized for speed at all. 
Therefore, a speedup by a factor of at least 10 ought to be possible using a "proper" 



'i.e. the computations proving that F 7 (x, 3/) is indeed of the specified type. 
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8 
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42 
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y 2 = x 10 + x 8 + 3x 6 + 4t 2 + 4 


4 


81.8 


F 3 


y 2 = x 9 + 2x 7 + 2x 3 + 2x 


8 


46.6 



Table 1. Time to compute Aut(k(x,y)/k) on an Intel® Cele- 
ron®, 1.7GHz, ordered by genus and |Aut(fc(x, y)/k)\ 



implementation. Nevertheless, the examples given in table 5 suggest that even this 
implementation computes the automorphism group Aut(fc(x, y)/k) of an arbitrary 
hyperelliptic function field very efficiently The performance seems to depend nei- 
ther on the size of the constant field, nor on the order of Aut(k(x,y)/k). Even 
though increasing the genus increases the size of the systems of polynomials — the 
number of both the polynomials and the parameters increase linear with g for types 
like F[C2, k] — , the examples indicate that even for fields of genus 4 and higher, the 
automorphism group computations are quite fast. 

Let us discuss the cryptographic application, briefly. As explained in the in- 
troduction, the initial goal was to provide an algorithm to check, whether a given 
hyperelliptic curve promises to yield a secure Jacobian, i.e. whether it is worthwhile 
to apply more expensive algorithms to check a given curve for security. Because of 
the attacks mentioned in the introduction, secure curves have small automorphism 
groups Aut(fc(x, y)/k). Since Aut(k(x,y)/k) < Aut(fc(x, y)/k), algorithm 7 can be 
used to assure this property. The timings of table 5 also apply to the set of relevant 
curves, as secure curves are of genus < 4 because of the Adleman-DeMarrais-Huang 
attack ([ADH94]) and as characteristic of the constant field and the size of the au- 
tomorphism group do not seem to influence the running time. 

Even though a small automorphism group is necessary for a secure curve, it is not 
obvious, how much information concerning security can be deduced from knowing 
the automorphism group. A discussion of this topic can be found in [Gobon]. 

The methods described in section 4.2 were not implemented. Nevertheless, we 
will try to compare algorithm 7 to Michael Stoll's AutomorphismGroup function (cf. 
[StoOl]) in some examples. To do so, we choose the smallest field k of the given 
characteristic, for which Aut(fc(x, y)/k) = Aut(k(x , y) / k) holds, in each example. 
Then, Aut(fc(x, y) / k) is computed using algorithm 7, while Stoll's method is used to 
compute Aut(k(x,y)/k). The running times for some examples are given in table 5. 

From these examples, Stoll's algorithm seems to be quite fast for small automor- 
phism groups, while it is very slow for large ones. As stated above, our implemen- 
tation does not seem to be influenced by the group size at all. Thus, if you are 
quite sure that the field you are investigating only has a small automorphism group, 
Stoll's algorithm ought to be preferred. Even though the majority of hyperelliptic 
function fields has a small automorphism group, the remaining fields do not seem 
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F 3 e y 2 = x 9 + 2x A + x + 2 


36 


2.9 27.7 


F52 y 2 = x b + Ax 


240 


18.1 22.7 


F 7 2 y 2 = x 7 + 6x 
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228.3 61.0 


F 3 4 y 2 = x 9 + 2x 


1440 


1347.3 34.1 


F U 2 y 2 = x 11 + lOx 


2640 


5625.1 90.3 5 



Table 2. Running time comparison between Michael Stoll's algo- 
rithm and algorithm 7, timings in seconds on an Intel® Celeron®, 



1.7 GHz 



to be suited for Stoll's algorithm. Hence, in order to compute the automorphism 
group of an arbitrary hypcrelliptic function field, it might be sensible to use the 
algorithms from section 4.2 as those at least seem to be more predictable w.r.t. 
performance. Furthermore, Stoll's algorithm returns every single automorphism, 
while the methods presented here, give the structure as well as the generators of 
the automorphism group. Thus, it also depends on the application, which of the 
algorithms ought to be used. 
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